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Generation of Mixed Media Streams 



FIELD OF INVENTION 



The present invention relates to the generation of mixed 
media streams from input media streams having payload data 
15 elements and related identifiers, and in particular 

considering a relation between different media streams 
during generation of a mixed media stream. 



20 BACKGROUND ART 

Mixed media streams are generated in communication 
networks, e.g., when mixing audio signals and video signals 
during a video conference. Here, it is important that mixed 
25 audio signals are matched to the related mixed video signal 
because otherwise the speech will not be lip -synchronous to 
the video stream. The same problem also arises with other 
streams than audio or video streams, e.g., text streams 
having alphanumeric signs when using subtitles. 
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Currently, the matching of such video and audio or in more 
general sense mixed media streams require a complicated 
procedure. Normally, time stamps are used, which will be 
attached to the different signals to enable the matching of 
5 the related media streams. However, while this at least is 
some mechanism to a match the mixing of different media 
streams, currently, there does not exist any solution to 
the problem how the generation of mixed media streams of a 
different type, e.g., a mixed video stream and a mixed 
10 audio stream, may be coordinated. 



SUMMARY OF INVENTION 

15 In view of the above, the object of the present invention 
is to provide a mechanism for coordination during 
generation of a plurality of mixed media streams. 

According to the present invention, this object is achieved 
20 through a method of generating a mixed media stream from 
input media streams having payload data elements and 
related identifiers, wherein the input media streams are 
aligned according to a pre-specif ied relation between 
identifiers in different input media streams before 
25 generating the mixed media stream. 

In the sense of the present invention a media stream is 
generated in a packet switched transmission process 
typically by a user end equipment . One example of an input 
30 media stream may be a voice data packet stream or a video 
data packet stream which is generated in a video 
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conference. Another example would be the exchange of media 
streams over any type of network, e.g., the Internet, an 
Intranet, an LAN, a WAN or VPN. 

5 Typically, each media stream consists of a plurality of 
data packets and each data packet has a header part and a 
payload data part. The header part carries identifiers 
characterizing a time relationship between different data 

packets in a single media stream. A single end user 
10 equipment generates different media streams - e.g., voice, 
data, text, sound etc. - and data packets of different type 
media streams carry the same identifiers when they are 
generated at the same instance of time. 

15 When different end user equipments generate different media 
streams of comparable type each such media streams for each 
end user equipment will then be split between different 
mixers in the communication systems handling the data 
packet exchange before mixing of similar type media streams 

20 into a mixed media stream and subsequent transmission over 
the communications network. 

The present invention considers the coordination of the 
operation for different mixers. 

25 

To achieve this, it is proposed to add a further step of 
alignment of different .input media streams in time before 
generating the mixed media stream. The alignment in time is 
achieved such that a pre- specif ied relative relation in 
30 time between different media streams is achieved. 
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Here, use is made of identifiers being available in the 
input media streams. As outlined above, in each media 
stream there are available identifiers defining an order of 
data packets in that media stream. The same also applies 
5 for all other input media streams which will finally be 
used to generate a specific mixed media stream. 

Therefore, at a certain point in time one may consider a 
tupel of such identifiers in each input media stream as a 
10 relation which may then be compared to a pre- specif ied 
relation. 

In other words, each such tupel defines a relative 
alignment of input media streams which may be changed 
15 according to a pre- specif ied relation either through 
advancing or delaying input media streams in time. 

Therefore, the result of the inventive method is a 
modification of the relative alignment of input media 
20 streams according to a pre-specif ied relation of ordering 
for the input media streams before generating the mixed 
media streams. It should be noted that such an alignment is 
related to the relative alignment of input media streams 
only and does not rely on absolute time. 

25 

According to a preferred embodiment of the present 
invention the pre -specif ied relation between identifiers in • 
different input media streams is matched to a relation 
between identifiers in further input media streams used 
30 during generation of a further mixed media stream. 
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Here, one example could be that the mixing of a first group 
of input media streams is related to speech data and the 
mixing of a second group of input media streams is related 
to video, text, service-related data, etc. Then, what is 
5 assured is that the generated first mixed media stream and 
the generated second mixed media will be generated using 
the same relative relation between identifiers in the 
related groups of input media streams* In other words, the 
generated mixed media streams are generated with the 
10 related input media streams being aligned according to the 

* 

same relative relation between identifiers. 



Here, a relation between identifiers in a first mixer - 
that may also be referred to a master mixer, without 
15 limiting scope of invention - may be used for the 

generation of a further mixed media stream in a second 
mixer - which may be referred to as slave mixer, without 
limiting scope of invention. 

20 It should be noted that according to the present invention 
each type of mixer in the communication network may be 
operated either as a master mixer or a slave mixer as long 
as it implements the inventive method. 



25 It should further be noted that according to the present 
invention a further alternative is that the relation 
between, identifiers from input media streams may be 
determined in a stand-alone alignment unit provided 
externally to mixers adapted to. generate mixed media 

30 streams. Here, the alignment unit would inform the mixers 
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about the relation between identifiers for siabsequent 
processing of input media streams in the mixers. 

According to a further preferred embodiment of the present 
5 invention, a matching of relations between identifiers in a 
first group of input media streams and in a further group 
of input media streams is achieved in an efficient way. 

Initially, an intersection between the input media streams 
10 and the further input media streams is identified. The 
purpose of this step is to tackle situations where not 
every mixer is operating on the same and related number of 
input media streams. 

15 One such situation may, e.g., occur if different user end 
equipments participating in a communication session 
generate different media streams, e.g., a first user end 
equipment may generate voice data only, a second user end 
equipment may generate video data only, and a third and 

2 0 fourth user end equipment may generate both voice and video 
data- In this case, only a coordination of the mixing for 
. media streams of the third and fourth user end equipment 
may make sense as only here a mutual relative relation 
between identifiers fixing a relative order data packets in 

25 a media stream is available. 

According to the preferred embodiment, once such an 
intersection is identified what follows is the 
determination of the relation between identifiers in input 
30 media streams at a master mixer for subsequent use during 
mixing of input media streams in a slave mixer. 
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In a further step, those input media streams in the slave 
mixer which are comprised in the identified intersection 
will be aligned according to the relation of identifiers of 
5 the input media streams handled by the master mixer. 

According to another preferred embodiment of the present 
invention the method is efficiently implemented through use 
of a set representation of input media streams . Each group 

10 input media stream is characterized by a set of tupels. A 
first element in each tupel characterizes an input media 
stream, e.g., through an input media stream number, and a 
second element in each tupel defines an identifier of the 
input media stream at a point in time. Thus, the 

15 determination of the intersection explained above may be 
achieved through simple set operations. 

* 

•A further preferred embodiment of the present invention is 
particularly suited to a situation where related input 
20 media streams in different groups of input media streams 
are not supplied to corresponding input terminals of 
different mixers. 

Here, one example would be that a first user end equipment 
25 generates a first voice media stream and a first media 

stream and a second user end equipment generates a second 
voice media stream and -a second video media stream. 
Nevertheless, a situation may occur where at a first mixer 
handling voice media streams the first and second voice 
30 ■ media streams are supplied to the first and second input 
terminal, while at a second mixer handling video media 
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Streams the first video media stream and the second video 
media stream are supplied to the second and first input 
terminal, respectively, or in other words, are 
interchanged . 

5 

In this case, it is very useful' to re-order the sequence of 
input media streams and the input terminals either at the 
master mixer or at the slave mixer because otherwise the 
alignment procedure outlined above will lead to a non- 
10 intended result. Preferably, such a re-ordering may be 

described using a permutation vector which easily allows to 
extend the set operations described above for incorporation 
of the re-ordering into the processing of different groups 
of input media streams. 



Another preferred embodiment of the present invention 
relates to the way the alignment of input media streams is 
achieved. 



20 An important aspect of the alignment is the determination 
of a time delay for each input stream such that - assuming 
that the time delay is subsequently applied to the input 
media stream - the relations between the identifiers in the 
input media streams then correspond to the pre- specif ied 

25 relation of identifiers. 

Also, the .time delay may be achieved through shifting each 
input media stream according to the related time delay, 
e.g., using a buffer memory of suitable size. 

30 
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Yet another preferred embodiment of the present invention 
is related to the exchange of information regarding the 
pre-specif ied relation between identifiers of different 
input media streams . 

A first way to achieve this exchange is through forwarding 
a related signal from the master mixer to the slave mixer 
which signal should comprise at least the relation of 
identifiers used during generation of a mixed media stream 
in the master mixer. 

Another mechanism suitable for the exchange of tupel 
between a master mixer and a slave mixer may be a shared 
memoiy provided between a master mixer and the slave mixer, 
^here each mixer has access to the shared memory in a time 
coordinated manner. 

Similar advantages as outlined above with respect to the 
inventive method are also achieved by an inventive 
20 apparatus for generating a mixed media stream from input 
media streams having payload data elements and related 
identifiers and preferred embodiments thereof. 

Further, according to yet another preferred embodiment of 
25 the present invention there is provided a computer program 
product directly loadable into the internal memory of a 
media mixer comprising -software code portions for 
performing the inventive mixing process when the product is 
run on the processor of the media stream processor. 
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Therefore, the present invention is also provided to 
achieve an implementation of the inventive method steps on 
computer or processor systems. In conclusion, such 
implementation leads to the provision of computer program 
products for use with the computer system or more 
specifically a processor comprised in, e.g., a media stream 
mixer . 

This programs defining the functions of the present 
invention can be delivered to a computer/processor in many 

■ 

forms, including, but not limited to information 
permanently stored on non-writable storage media, e.g., 
read only memory devices such as ROM or CD ROM discs 
readable by processors or computer I/O attachments; 
information stored on writable storage media, i.e. floppy 
discs and harddrives; or information convey to a 
computer/processor through communication media such as 
network and/or telephone networks and/or Internet via 
modems or other interface devices. It should be understood 
that such media, when carrying processor readable 
instructions implementing the inventive concept represent 
alternate embodiments of the present invention. 

DESCRIPTION OF DRAWINGS 

In the following preferred embodiments of the present 
invention will be described with reference to the drawings 
in which: 
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Fig. 1 shows a schematic diagram of a media stream mixer 

according to the present inventions- 
Fig . 2 shows a flowchart illustrating the operation of 

the media stream mixer according to the present 
invention; 

Fig. 3 shows a schematic diagram of the alignment unit 

shown in Fig. 1; 

Fig. 4 shows a flowchart illustrating the operation of 

the alignment unit shown in Fig. 3; 

Fig. 5 shows a first example for the alignment of media 

streams according to the present invention; 

Fig. 6 shows a second example for the alignment of media 

streams according to the present invention; and 

Fig. 7 shows a third example for the alignment of media 

streams according to the present invention. 

DESCRIPTION OP BEST MODE AND PREFERRED EMBODIMENTS 
OF THE INVENTION 

In the fol-lowing, general reference will be made to the 
mixing of media streams. Here, it is assumed that each 
media stream is set up by a sequence of data packets 
dividing into a header part and a payload part. The header 
part will comprise identifiers defining a sequential 
ordering of the data packets. Media stream mixing in the 
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sense of the present invention is related to the reception 
of a plurality of media streams of a common type and the 
processing of the plurality of media streams such that only 
a single media stream of a mixed type according to a 
specific processing on the input media streams is output. 
The present invention is well suited for any type of 
processing on input media streams, e.g., audio streams, 
video streams, and/or text streams. In other words,, the 
mixing of media streams is not restricted to any type of 
media streams, e.g., voice, data, text, etc. 

Fig. 1 shows a schematic diagram of a media stream mixer 
according to the present invention. 

As shown in Fig. 1, the media stream mixer 10 comprises a 
stream input unit 12 adapted to receive at least one input 
media stream, a stream processing unit 14 generating a 
single output media stream from the at least one input 
media stream, and a stream output unit 16 for output of the 
generated mixed media stream. 

Further, the media stream mixer 10 comprises a memory 18 
and an interface unit 20 for exchange of identifier 
information to be used in the stream processing unit. The 
stream processing unit 14 comprises an alignment unit 22, a 
mixer unit 24, and an identifier unit 26. While in Fig. 1 
the alignment unit is shown in relation to the stream 
processing unit 14, it, should be noted that according to 
the present invention it is also possible to provide the 
alignment unit externally to the stream processing unit 14, 
e.g., in the stream input unit 12 or as stand-alone unit 
outside the media stream mixer. 



wo 03/105484 




PCT/EP02/06400 



Fig. 2 shows a flowchart illustrating the operation of the 
media stream mixer according to the present invention. 

As shown in Fig. 2, in a first operation step SIO the media 
stream mixer will receive at least one input media stream 
of a pre- specif ied type. Then, in a step S12 the media 
stream mixer will align input media streams according to a 
pre- specified relation, and in a step S14 the media stream 
mixer will output the aligned media stream. 

Heretofore, the stream processing unit 14 of the media 
stream mixer will exchange identifiers defining a pre- 
specified relation to be achieved for the input media 
stream before mixing thereof, by the identifier interface 
20. 

For comparison of a pre-specif ied relation of identifiers 
with the actual relation of identifiers in different input 
media streams, the stream processing unit 14 comprises the 
identifier unit 26 adapted to extract identifiers from 
input media streams. 

The step of alignment S12 shown in Fig 2 is achieved by the 
alignment unit 22 shown in Fig. 1, and the mixing of the 
input media streams of alignment is achieved by the mixer 
unit 24 . 

> • 

t > 
I 

In the following, a more detailed explanation of the 
alignment of input media streams according to the present . 
invention will be given with respect to Fig. 3 and 4. 
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Fig. 3 shows a schematic diagram of the alignment unit 22 
shown in Fig. 1. As shown in Fig. 3, the alignment unit 
comprises a pre-processing unit 28, a stream shift 
calculation unit 30 and a stream shift unit 32. 

Operativelx, the pre-processing unit 2 8 is adapted to 
identify which input media streams of the plurality of 
input media streams must be aligned according to a pre- 
specified relation. 

The stream shift calculation unit 30 is adapted to 
determine a correct ordering of the sequence of input media 
streams. Further, once relevant input media streams are 
brought into a proper order, or in other words, into the 
same order as input media streams considered for defining 
the pre -specified relation between identifiers available in 
the master mixer - the stream shift calculation unit 30 
will then proceed with the calculation of an appropriate 
shift to achieve the required alignment of input media 
streams in the stream shifting unit 32. 

Finally, the stream shifting unit 32 will achieve an 
alignment of input media streams according to the result 
determined by the stream shift calculation unit 30, e.g., 
through intermediate storage of specific input media 
streams in the memory 18 of the media stream mixer 10. 

The operation of the alignment unit shown in Fig. 3 may be 
summarized according to a flowchart illustrating the 
operation as shown in Fig. 4, 
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As shown in Fig. 4 the alignment unit 22 basically 
identifies which input media streams to the mixed media 
mixer must be processed in a step S20. Then, it determines 
an appropriate time delay with those media streams which 
have to be processed in a step S22, In a final step S22 it 
will shift media streams to be processed according to the 
determined time delay in a step S24, before subsequent 
mixing thereof. 

In the following, the operation of the media stream mixer 
illustrated with respect to Fig. 1 to 4 will be described 
on the basis of a more formal notion. 

Heretofore, it may be assumed that a first mixer is 
considered as a master mixer indicated with index ^ and a 
second mixer is considered as slave mixer indicated with 
index g . 

The set of media streams handled by the master mixer may be 
described through a set of tupel Si, ii, where Si relates 
to a number of each input media stream and ii relates to an 
identifier of this stream si at a point in time which 
changes during operation on the set of input streams. 

Similarly, for the slave mixer the set of input media 
streams at a point in time is described by a set of tupel 
the qi is related to the number of a specific input media 
stream and related ni is the identifier comprised in the 
input media stream at a point in time: 
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5 Here, one example may be that a master mixer operates on a 
set of input media streams 2, 4, 5, which at a point in 
time carry identifiers 1, 4, and 7. Further, for 
illustrating purposes one may assume that the slave mixer 
operates on a set of input media streams 2, 5, 4, 10 which 
10 at a point" in time carry the identifiers 2, 1, 5, 2. 



In view of the examples given above, the present invention 
now considers how the ordering and time relationship 
between the different input media streams at the slave 
mixer may be modified such that they correspond to the 
20 ordering' the time relationship between the input media 
streams and the master mixer. 

Here, it should be clear that a modification of input media 
streams is only achievable in the slave mixer for those 
25 input media streams which have a counter part in the master 
mixer. .Therefore, a first step is the determination of a 
subset of input media streams at the master mixer and slave 
mixer which are coincident according to 



Im 4(2,1), (4,4),(5,7)] 



Is =[(2,2),(5,1),(4,5),(10,2)] 
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30 ={Ji,52v,'S„}n{?i,52,.-.,e,} 
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L5 



For the particular example given above the result would be 
the input media streams having the number 2, 4, 5, 
respectively : 

= {2,4,5} 

In view of the determined intersection of input media 
streams at the master mixer and the slave mixer, one may 
then modify the set of input media streams at the master 
mixer and the slave mixer to be in compliance with the 
intersection according to 



1^'= U {(^p^OU.-e-Sj 



which for example leads to 



20 /«H(2,1),(4,4),(5,7)] 



'=[(2,2), (5,1), (4.5)] 

From the example one may see that not only the sequence of 
25 the input media stream's must be considered, but also their 
ordering. This ordering is - according to the stream 
numbers - 2, 4, and 5 for the master mixer, but 2, 5, 4 for 
the slave mixer. This implies that it is not possible to 
carry out an immediate consideration of identifiers of the 



LO 



WO 03/105484 PCT/EP02/06400 

18 





input media streams at the master mixer and the slave mixer 
without appropriate modification of the input media stream 
ordering . 

5 Such a modification may be represented by a permutation 
vector. Each element of the permutation vector expresses 
which input media stream to the master mixer corresponds to 
the which input media stream of the slave mixer according 
to 



V {p(0 = ye[l,...l//|]|^,=9^} 



For the specific example referred to above, the result is 



15 p = [W] 

The different steps described so far in a formal manner and 
also in view of an example may be achieved in the pre- 
processing unit 28 shown in Fig. 3. 

20 

The further steps to be explained in the following are 
achieved by the stream shift calculation unit 30 also shown 
in this Fig. 3- 

25 The first step to be achieved by the stream shift 
calculation unit 30 is to modify the set of input streams 
to the slave mixer according to the permutation vector 
derived as outlined above. In other words, the set of input 
media streams to the slave mixer is brought into an 
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ordering according to the set of input media streams to the 
master mixer: 

//=0 

which for the example referred to above leads to 
// '=((2,2), (4,5), (5,1)] 

One should note that the modification of the set of input 
streams to the slave mixer must not be implemented through 
actual re-ordering of input media streams at the input 
terminals of the slave mixer but may be achieved through 
indirect access to the input media streams using the 
permutation vector, as will be explained in the following. 

Operatively, the stream shift calculation unit 30 
determines each identifier in the input media streams to 
the master mixer and compares it with the related 
identifiers of input media streams to the slave mixer. The 
comparison is based on 

Here, the selection of' the write input media stream at the 
slave mixer for comparison with the related input media 
stream to the master mixer is achieved through indirect 
access to the identifiers of the input media streams to the 
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slave mixer using the permutation vector. For the specific 
examples given above, the result of this step will be 

^ = [-1-1,6] 

As can be seen from the above, there may be input media, 
streams to the slave mixer which are either delayed or 
advanced in relation to the corresponding input media 
stream to the master mixer. 

However, as the stream shift unit 32, explained in more 
detail in the following, may only delay input media streams 
at the slave mixer, one has to take care that the final 
alignment consists solely of delays of input media streams 
to the slave mixer. 

According to the present invention, this is no problem as 
only a relative relation between input media streams must 
be maintained at the slave mixer in view of a relation of 
identifiers in input media streams at the master mixer 
without an absolute time scale. 

Therefore, one will modify the delay vector generated 
through comparison of identifiers in the input media 
streams to the master mixer and the slave mixer according 
to 

< 
■ 

S/=S;- max = 1> .-,1 /^'l 



For the example referred to above, this leads to 
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S'^ [-7,^7,0] 

As can be seen, after this step only delays for the input 
media streams to the slave mixer occur. These delays are 
realized through intermediate storage of data packets 
comprised in each of the input media streams to the slave 
mixer . 

In other words, the delay vector generated in the stream 
shift calculation unit 30 will be used by the stream 
shifting unit 32 such that the input media streams to the 
slave mixer are delayed according to 

(/,.'(0=g..(^ + '^/') 

which for the example referred to above leads to 
(?4'(0 = 94('-'7) 

It should be noted that the formulas specified above may be 
modified in reversing the signs and the maximum operator to 
the minimum operator which would lead to the same result. 

. . • 

Fig. 5 shows a first exaitple for the alignment of media 
streams according to the present invention. 

As shown in Fig. 5, according to a first example, three 
user end terminals 34, 36, and 38 generate simultaneously 
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two types of media streams 40, 42, 44, 46, and 48, 50. 
Here, according to the exan5)le shown in Fig. 5, the media 
streams 40, 44, and 48 are of similar type, and also the 
media stream 42, 46, 50 may be of similar type. Similar 
5 media streams are supplied to a related mixer via splitters 
52, 54, and 56. 

In more detail, these splitters 52, 54, 56 submit the media 
streams 40, 44, 48 to a master mixer and the media streams 
LO 42, 46, 50 to a slave mixer 60. As outlined above, each 
master mixer 58 and slave mixer 60 generates a single 
output media stream 62 and 64 which may then be forwarded 
to the receiving equipment 66. 

15 It should be noted that each such generated output media 
stream and the related data packets comprised therein are 
identified by a master ID. 

According to the present invention, the relation between 
20 identifiers in input media streams at the master mixer 58 - 
e.g., i6, i22, il3, and the master ID will then be 
forwarded at specific points in time - either periodically 
or not - to the slave mixer 60 via a signalling line. 
. Although not shown in Fig. 5, an alternative mechanism for 
25 data exchange between the master mixer 58 and the slave 
mixer 60 may be a shared memory type communication. 

( 

Fig. 6 shows a second example of the alignment of media 
streams according to the present invention. 

30 
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The example shown in Fig. 6 is similar to the example shown 
in Fig. 6, so that the same reference numerals are used to 
denote the same structural elements. In the example shown 
in Fig. 6, data packets of input media streams and output 
5 media streams are shown for one point in time. 

According to the example shown in Fig. 6, each end user 
equipment 34, 36, 38 generates a video media stream and an 
audio media stream which are distributed via splitters 52, 
LO 54, 56 to a video mixer 58 operating as master mixer and an 
audio mixer operating as slave mixer. 

The time relationship between the video input media streams 
and their related data packets at the point in time shown 
L5 in Fig. 6 is (12, 23, 42) and is exchanged from the video 
mixer 58 to the audio mixer 60. 

■ 

Further, the audio mixer will receive the master • ID=1 of 
the data packet generated by the video mixer 58 for 
20 forwarding to the receiving equipment 66. The audio mixer 
60 will receive the information from the video mixer and 
combine data packets having identifiers (12, 23, 42) also 
for the audio input media streams and generate a related 
output data packet under the same master ID. 



25 



30 



Therefore, at the audio mixer not only the relative time 
relationship between data packets in video input media 
stream used at the video mixer is considered for generation 
of an output data packet of the audio type, but this output 
data packet is also characterized by the same master ID, so 
that at the receiving equipment 66, both the mixed video 
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data packet and the mixed audio data packet are referenced 
under the correct master ID. Therefore, video display and 
audio display will be in proper time relationship. 

5 Fig. 7 shows a third example of the alignment of the media 
streams according to the present invention. 

The example shown in Fig. 7 is different over the previous 
examples in that the number of input media streams handled 

.0 by the master mixer and the slave mixer is different. 
Further, the input media streams at the second and third 
input terminal of the master mixer and the slave mixer are 
interchanged. Finally, for this example it is assumed that 
also the mutual relation between input media streams at the 

.5 master mixer is different from that of the slave mixer. 

As shown in Fig. 7, to achieve the same relative relation 
between identifiers (1, 4. 7), also at the slave mixer it 
will be necessary to apply an intermediate storage for the 

20 input media streams supplied to input terminal q2 of the 
slave mixer. In other words, as shown on the right side of 
Fig. 7, at the slave mixer at input terminal 1 and 3, there 
are supplied input media streams 2, 4 according the input 
media streams supplied to the first and second input 

25 terminals of the master mixer. 

Further > the difference between the related identifiers (5 
- 2 = 3) is similar to those of the related input media 
streams at the master mixer (4-1=3). 

30 



wo 03/105484 




25 




PCT/EP02/06400 



However, the input media stream supplied to the second 
input terminal of the slave mixer, i.e. input media stream 
No. 5 corresponding to the input media stream supplied to 
the third input terminal of the master mixer is not aligned 
5 properly . 

To achieve the correct alignment, as shown in the lower 
table of Fig. 7, the input media stream to the first and 
third terminal of the slave mixer - i.e. the input media 
.0 stream with Nos. 2 and 4 - should be supplied to an 
intermediate storage, while the processing of the input 
media stream to the second input terminal of the slave 
mixer is continued without delay. 

L5 After processing of seven such data packets being related 
to the input media stream to the second input terminal of 
the slave mixer, the related identifier in the input data 
packet will be (8) . The difference between the input media 
stream being supplied to the second input terminal of the 

20 slave mixer and the input media stream being supplied to 
the third input terminal of the slave mixer will be (8 - 5 
= 3) . This difference is in compliance with the difference 
between identifiers in the input media streams supplied to 
the second and third input terminal of the master mixer (7 

25 -4 = 3). 

At this instance of time, it is then possible to combine 
the buffered data packets of the input media streams with 
Nos. 2 and 4 and supplied to the first and third input 
30 terminal of the slave mixer with the input media streams 
being supplied to the second input terminal of the slave 
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mixer and being processed without buffering, as shovna with 
the shaded rectangles in Fig. 7. 

According to the present invention, it is also suggested as 
5 option to maintain a relation of the input media stream 
supplied to the fourth input terminal of the slave mixer 
during subsequent processing of all input media streams to 
the slave mixer. 



While above reference has been made to identifiers in input 
media streams, generally, examples of such identifiers 
could be time stamps or sequence numbers. Also, the 
exchange of a relation such identifiers from the master 
mixer to the slave mixer may be achieved periodically when 
15 sending a SYNC signal to slave mixers. Here, one may either 
send all identifiers being handled in the master mixer or 
information about an absolute identifier value and the 
relation of identifiers, e.g., IDl = x, ID2 = IDl + y, ID3 
= ID + z, etc. 



Further, the data exchange between the master mixer and the 
slave mixer has almost no delay requirement and the 
periocity may not be very high, assuming a reasonable low 
drift between the clock sources in the different mixers. 
25 The exchange signal, e.g., the SYNC signal, may be sent in 
different ways, as timing requirements are not very high. 
Example- implementations- would be out-band signalling, in- 
band signalling or via an operating system. 
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Claims 



Method of generating a mixed media stream from input 
media streams having payload data elements and related 
identifiers, comprising the step: 

aligning the input media streams according to a 
pre- specif ied relation between identifiers in 
different input media streams before generating 
the mixed media stream. 

Method according to claim 1, characterized in that the 
pre- specif ied relation between identifiers in 
different input media streams is matched to a relation 
between identifiers in further input media streams 
used during generation of a further mixed media 
stream. 

Method according to claim 2, characterized in that the 

matching of relations between identifiers in the input 
media streams and in the further input media streams 
is achieved by- 
identifying an intersection between the input 
media streams and the further input media 
streams, 

determining a relation between identifiers in the 
further input media streams for those further 
input media streams which are comprised in the 
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intersection, 

aligning the input media streams which are 
comprised in the intersection according to the 
relation of identifiers in the further input 
media strems. 

Method according to claim 3, characterized in that the 
input media streams at a point in time are described 
by ^5 ={(9i."i);(92>«2X' with {gpg2» -i9m} as set of 
input media streams and {«i,«2» set of 

identifiers in the input media streams at a point in 
time, 

the further input media streams at a point in time are 
described by 1^ =iis^Ji)Xs2^i2)r-Jis^,iJ] with {s,,S2,.,.,sJ as 
set of further input media streams and {ip^jj • -jC) 
set of identifiers in the further input media streams 
at the point in time, 

the intersection between the input media streams and 
the further input media streams is 

= i^i^^zr-'psj r^{qi,q2,,..,q^} , and 

the relation between identifiers in the further input 
media streams for those further input media streams 
which are comprised in the intersection is determined 
on the basis of I/^ U 1 9- ei^.} • 

■ I 

I 

Method according to claim 4, characterized in that it 

further comprises a step of re-ordering the sequence 
of input media streams in the intersection according 



■ 
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to the sequence of further input media streams in the 
intersection. 

6. Method according to claim 5, characterized in that the 

re-ordering of the sequence of input media streams is 
achieved according to a permutation vector defined 



7. Method according to one of the claims 3 to 6, 

LO characterized in that the alignment of input media 

streams in the intersection is achieved by- 
determining a time delay for each input media 
stream such that relations between different 
15 identifiers in the input media streams after time 

delay correspond to relations between identifiers 
in the further input media streams, and 

shifting each input media stream in time 
20 according to the related time delay. 

8. Method according to claim 7, characterized in that the 

time delay for each input media stream is determined 
according to 



Si'^Sj- max (J,,/ = 1,...,|//| 



30 



and each input media stream is shifted in time 
according to 
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(7,;(0=?..(^+^/V=u.i//i. 

9. Method according to one of the claims 1 to 8, 
characterized in that an exchange of information 

regarding the pre- specif ied relation between 
identifiers is achieved through a signal transfer or a 
shared-memory mechanism - 

10 . Apparatus for generating a mixed media stream from 
input media streams having payload data elements and 
related identifiers, comprising: 

an alignment unit (22) adapted to align the input 
media streams according to a pre -specif ied 
relation between identifiers in different input 
media streams before generating the mixed media 
stream. 

11. Apparatus according to claim 10, characterized in that 

it comprises an identifier interface unit (20) for 
exchange of the pre- specif ied relation between 
identifiers in different input media streams. 

12. Apparatus according to claim 10 or 11, characterized 
in that the pre- specif ied relation is matched to a 

relation between identifiers in further input media 
streams used during generation of a further mixed 
media stream and that the matching of relations 
between identifiers in the input media streams and in 
the further input media streams is achieved by 
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a pre-processing utnit (28) adapted to identify an 
intersection between the input media streams and 
the further input media streams, 

5 

"a calculation unit (30) adapted to determine 
a relation between identifiers in the 
further input media streams for those 
further input media streams which are 
.0 comprised in the intersection, and 

the aligning unit (32) adapted to align the 
input media streams which are comprised in 
the intersection according to the relation 
-5 of identifiers in the further input media 

streams . 

13, Apparatus according to claim 12, characterized in that 

20 the input media streams at a point in time are 

described by =[(?p"i),(?2>«2)> ••^(9,>'^J] with {?i,92> ->?in} as . 
set of input media streams and {n^^n^i-.-.^n^) as set of 

identifiers in the input media streams at a point in 
time, 

25 

the further input media streams at a point in time are 

described by Ij^ =[(^i>0.('^2>^2)r-X'^m>^m)] ^^th {^p^zv-^-y^} as 
set of further input media streams and ftji2''"-»'m} as 

set of identifiers in the further input media streams 
30 at the point in time, 
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the pre-processing unit (28) is adapted to determine 
the intersection between the input media streams and 
the further input media streams according to 

the pre-processing unit (28) is further adapted to 
determine the relation between identifiers in the 
further input media streams for those further input 
0 media streams which are comprised in the intersection 

on the basis of I/^ U {(?/>«/) 1 9,- • 

^€11,...^] 

14. Apparatus according to claim 12 or 13, characterized 
in that the pre-processing unit (28) is further 

.5 adapted to re-order the sequence of input media 

streams in the intersection according to the sequence 
of further input media streams in the intersection. 

15. Apparatus according to claim 14, characterized in that 

the pre-processing unit (28) is adapted to re-order 
the sequence of input media streams according to a 
permutation vector defined by 

. V,^.,M0 = ye[l.-.|/5'l]U, =9;}. 

16. Apparatus according to one of the claims 12 to 15, 
charapterized in that the alignment of input media 

streams in the intersection is achieved by 

the calculation unit (30) adapted to detemine a 
30 time delay for each input media stream such that 



25 
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relations between different identifiers in the 
input media streams after time delay correspond 
to relations between identifiers in the further 
input media streams, and 

a shifting unit (32) adapted to shift each input 
media stream in time according to the related 
time delay. 

17. Apparatus according to claim 16, characterized in that 

the calculation unit (30) is adapted to calculate the 
time delay for each input media stream according to 

Si = If - ^p(i)j - I 

S/=Si- max S = ly>,.y\I/\ 

■ 

and the shifting unit (32) is adapted to shift each 
input media stream in time according to 

18. Apparatus according to one of the claims 11 to 17, 
characterized in that it comprises an identifier unit 

(26) adapted to identify a relation between 
identifiers in different input media streams at a 
point- in time and 'that the identifier interface (26) 
is adapted to forward at least two identifiers in the 
input media streams to a, further apparatus for 
generating a mixed media stream for use during 
generation of the further mixed media stream. 
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19. A computer program product directly loadable into the 
internal memory of a mixer unit comprising software 
code portions for performing the steps of one of the 
5 claims 1 to 9 when the product is run on a processor 

of the mixer unit . 
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